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ABSTRACT ARTICLE HISTORY 
Since the 2000s, states have experimented with reforms to Received 19 November 2019 
improve success among underprepared students traditionally Accepted 10 June 2020 
assigned to developmental education (DE). Florida’s reform KEYWORDS 

under Senate Bill 1720 has been among the most comprehen- Developmental education; 
sive and wide-reaching. Recent public high school graduates state policy; student success 
and military personnel became exempt from DE, but nearly one- 

third of students, including those without a Florida standard 

high school diploma, were still required to take a placement test 

and enroll in DE if they scored below college-ready. The legisla- 

tion also required colleges to offer accelerated instructional 

strategies for students remaining in DE, and provide enhanced 

advising and support services. Focusing specifically on non- 

exempt students, we use statewide data to conduct 

a difference-in-regression discontinuity analysis to examine dif- 

ferences in first-year math coursetaking outcomes for students 

on the margins of college readiness before and after the reform. 

While students narrowly assigned to DE tend to have a lower 

likelihood of taking and passing college-level courses relative to 

their college-ready peers, these students experienced larger 

gains after the reform when DE courses were offered in accel- 

erated formats accompanied by support services. The reform 

also improved outcomes for students scoring above college- 

ready, which suggests that nonexempt students benefited from 

enhanced advising and support services too. 


Students at community colleges often face a complex web of challenges invol- 
ving inadequate academic preparation, competing family and work obliga- 
tions, financial barriers, and difficulties navigating complex college systems 
and requirements (Bettinger et al., 2013). These challenges may be further 
exacerbated by difficulties in the classroom, thus increasing the risk of early 
departure prior to credential completion. Developmental education courses 
are widely used by community colleges to help support underprepared stu- 
dents from a wide variety of backgrounds, including recent high school 
graduates with inadequate academic preparation as well as returning adults 
who have been away from the classroom for many years. Yet developmental 
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education has been criticized for failing to sufficiently develop skills among 
underprepared students and discouraging persistence among students poten- 
tially misassigned to these courses who likely would have succeeded in college- 
level courses (Scott-Clayton & Rodriguez, 2015). 

In traditional developmental education programs, incoming college stu- 
dents take a placement test and these scores are used to place students into one 
or more levels of developmental education in reading, writing, and/or mathe- 
matics. In the mid- to late 2000s, the first wave of developmental education 
reforms began which included isolated practices such as providing additional 
tutoring or supplemental instruction, and implementing student success 
courses (Jaggars & Bickerstaff, 2018). These early reforms were intended to 
improve students’ progression to and success in introductory college-level 
courses, and focused on short-term interventions which tended to be small 
in scope. The second wave of developmental education reforms began around 
2010 and consisted of more wide-scale adoption of practices including more 
comprehensive placement strategies, the use of accelerated options such as 
compressed courses to allow for faster completion of developmental education 
requirements, and changes to the pedagogy and content of developmental 
education courses. Many of these second-wave reforms have demonstrated 
positive impacts on short-term outcomes like taking and passing college-level 
courses, but there are still challenges to improving longer-term outcomes like 
graduation rates for students who enter college underprepared (Jaggars & 
Bickerstaff, 2018). 

In 2014, Florida implemented one of the most comprehensive developmen- 
tal education reform efforts under Senate Bill (SB) 1720. This reform required 
all 28 institutions in the Florida College System (FCS) to make three significant 
changes simultaneously: (a) remove requirements for placement testing and 
developmental education courses for the exempt students, (b) offer different 
instructional strategies for students remaining in developmental education, 
and (c) provide additional advising and academic support services to all 
incoming students, regardless of their initial course placement. The first 
component of the reform that made developmental educational optional for 
the majority of students was the most controversial, as critics feared that it 
would harm those students at the lowest levels of preparation (e.g., Flannery, 
2014; Tierney & Duncheon, 2013). Yet Florida’s reform did not merely 
eliminate developmental education, rather it gave students the option of 
enrolling directly into a college-level course if they believed they were ready, 
or the option of enrolling in a developmental education course using new 
instructional modalities if they thought that they needed additional prepara- 
tion. These developmental courses were designed in a way to accelerate 
students’ progression to college-level coursework among both exempt stu- 
dents opting into developmental education and nonexempt students scoring 
below college-ready who were still required to take these courses. Colleges 
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were required to offer options for modularized, compressed, co-requisite, or 
contextualized developmental courses that could often be completed more 
quickly than traditional semester-longer courses. The additional advising 
services were intended to help students make good decisions about which 
courses to take, while the enhanced academic support services were designed 
to provide students with extra help so that they could succeed in their courses. 

However, not all students could take advantage of the option of enrolling 
directly into college-level courses. Students who entered a Florida public school 
in 2003/04 or later and completed a standard high school diploma, as well as 
activity duty military personnel, became exempt from developmental education. 
In contrast, students who had not earned Florida standard high school diplomas 
in this timeframe were still required to take a placement test and enroll in one to 
two developmental courses in math, reading and/or writing, depending on their 
test scores. The years selected for the exemption criteria coincide with increased 
academic standards in Florida’s public schools following the implementation of 
Governor Jeb Bush’s “A+ Plan” in 1999. Therefore, the exemption criteria were 
likely intended to address concerns about students entering college without 
secondary preparation under the reformed Florida public schools. 

Nonexempt students make up nearly one-third of all first-time-in-college 
(FTIC) students and include students from out-of-state schools, private 
schools, home schools, older students who attended public schools under 
prior state standards, and those who did not complete a standard high school 
diploma. Given the large number of students affected by these nonexemption 
criteria, it is important to examine how this population may have been affected 
by the reform. Additionally, the results provide context for understanding 
whether the benefits of Florida’s reform may be attributed solely to removing 
mandatory assignment to developmental education for exempt students, or 
whether there may also be benefits to changing the ways in which develop- 
mental education is offered for nonexempt students. Specifically, we address 
the following research questions: 


(1) How do first-year coursetaking outcomes differ based on assignment to 
developmental education for nonexempt students in the pre-policy 
years (under traditional developmental education) and in the post- 
policy years (under reformed developmental education)? 

(2) What is the effect of the developmental education reform on first-year 
coursetaking outcomes for nonexempt students? 

(3) Does the effect of the reform differ depending on nonexempt students’ 
assignment to developmental education? 


Prior research indicates the reform has led to an overall increase in student 
success as measured by completion rates in the first college-level course and 
college-level credit hours attempted and earned during the first year of 
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enrollment among all first-time-in-college students (Hu et al., 2019; Park- 
Gaghan et al., 2020). Further, examining the effectiveness of Florida’s devel- 
opmental education reform among the population of nonexempt students 
allows us to extend this research in several important ways. First, it allows us 
to examine whether concerns were valid that nonexempt students not com- 
pleting standard high school diplomas under more rigorous Florida state 
standards may not be adequately prepared to enroll directly into college- 
level coursework. Second, since nonexempt students were still required to 
take a placement exam, we can use more rigorous methods through 
a difference-in-regression discontinuity (DiRD) design to compare first-year 
coursetaking outcomes before and after the reform for students narrowly 
assigned to developmental education courses relative to those who narrowly 
avoided placement in these courses. Students scoring just below the college- 
ready cutoff would be subject to the greatest changes under the reform, as they 
would switch from placement in a traditional semester-long developmental 
education course to placement in an accelerated developmental course with 
additional support services. These new instructional strategies for develop- 
mental education went into effect after the reform, so comparing outcomes 
among students below this cutoff before and after the reform allows us to see if 
the effectiveness of developmental education changes. We can also examine 
whether the effects of the reform differed for students above or below the 
cutoff. The students scoring just above the cutoff would only experience 
a change in the support services provided after the reform while continuing 
to enroll in the first college-level math course. The students scoring below the 
cutoff would experience changes in instructional strategies in developmental 
education courses and the change in the support services. Using a rigorous 
empirical approach allows us to begin to tease out the extent to which out- 
comes may have changed due to enhanced advising and support services alone 
(based on students above the cutoff), relative to also being assigned to take 
a developmental education course under new accelerated instructional strate- 
gies (based on students below the cutoff). We focus specifically on placement 
in math, an area of critical importance given that math is the most commonly 
needed developmental education course and tends to have the lowest success 
rates, making it a “gatekeeper” to higher education success (e.g., Bryk & 
Treisman, 2010; Huang, 2018). We also examine whether the effects differ 
by academic preparation depending on whether students are near the cutoff 
for assignment to one or two levels of developmental math courses. 

The next section describes previous literature examining the impacts of both 
traditional and reformed developmental education courses on first-year course- 
taking outcomes, as well as how the results may differ for various subgroups of 
students such as lower-performing students and older adults. Next, we describe 
our analytic approach that employs a difference-in-regression discontinuity 
analysis to examine differences in first-year coursetaking outcomes for 
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nonexempt students on the margins of college readiness before and after the 
reform. Our data uses longitudinal student records from the population of 
nonexempt students at all 28 FCS institutions (the former community colleges). 
Lastly, we conclude with a discussion of the implications for this unique student 
population, and important considerations for policymakers in Florida and other 
states to consider when determining who may benefit from being allowed to opt- 
out of developmental education. 


Literature review 


Much of the rigorous research to-date on the impact of developmental educa- 
tion has been conducted using regression discontinuity (RD) designs to 
compare postsecondary outcomes for students who score just below-college 
ready and are assigned to developmental education courses relative to students 
who score just above-college ready and are assigned to college-level courses. 
Valentine et al. (2017) conducted a meta-analysis of studies using RD to 
examine the effects of placement into developmental education and found 
statistically significant negative effects on several postsecondary outcomes, and 
the effect sizes were often quite large in magnitude. In particular, students 
assigned to developmental education were approximately 8 percentage points 
less likely to ever complete a college-level course in the same subject area in 
which remediation was needed compared to similar peers who were college- 
ready. However, a major limitation of these RD studies is that the estimates are 
only for students on the margins of college readiness, and may not be general- 
izable to students with varying levels of academic preparation. 

To further explore this issue of whether the effects of developmental 
education differ by students’ levels of preparation, several studies have exam- 
ined the impacts for lower-performing students who are placed into more than 
one level of developmental education. One study by Dadger (2012) found that 
students in Virginia community colleges who were assigned to three develop- 
mental math courses would have been more likely to complete a certificate or 
associate’s degree if they had been allowed to take only two developmental 
math courses. Another study using data from four California community 
colleges found similar results, as students assigned to an extended two- 
semester sequence of developmental math were significantly less likely to 
complete subsequent courses and earned fewer degree-applicable credits rela- 
tive to students in a single semester-long developmental math course (Ngo & 
Kosiewicz, 2017). A third study by Melguizo et al. (2016) examined placement 
in four distinct levels of math in a large urban community college system and 
found that students assigned to lower level math courses had a lower like- 
lihood of progressing to the next course in the sequence relative to students 
placed in higher-level courses. Only one study by Boatman and Long (2018) 
has demonstrated more positive evidence of the impacts of developmental 
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education for lower-performing students assigned to multiple levels of devel- 
opmental courses. While there were negative, and often large, effects for 
students assigned to a single level of developmental education, most effects 
were null for students assigned to multiple levels of developmental math and 
there were some positive effects for students assigned to multiple levels of 
reading or writing. 

While we are not aware of any studies that have examined differences in 
the effects of developmental education by the type of high school attended 
(public, private, homeschool, in-state, or out-of-state), there has been some 
research on the subgroup of adult undergraduates (typically 25 or older) who 
have been out of high school for a number of years. Older adults are more 
likely to be placed into developmental education courses upon college 
enrollment and tend to be less likely to complete the developmental sequence 
to which they are assigned relative to traditional college-age students (Bailey 
et al., 2010). On one hand, they may be at a disadvantage because they are 
more likely to have additional work and family obligations than their 
younger peers, are more likely to enroll part-time, and may be more likely 
to have forgotten math skills learned in high school (Bettinger et al., 2013). 
Yet, on the other hand, they may have higher levels of motivation because 
they often return to school due to life transitions, and they may have a more 
realistic understanding of their skills based on their work experience. 
Calcagno et al. (2007) examined postsecondary outcomes for students 25 
and older in Florida two-year colleges. Compared to their younger counter- 
parts, these students were more likely to have obtained an alternative sec- 
ondary credential like a GED rather than a standard high school diploma and 
tended to score lower on the math placement exam upon college entry. The 
older students were less likely to complete a degree overall, but this trend 
reversed when taking into account differences in incoming math achieve- 
ment levels, suggesting that “old dogs can learn new tricks—after they 
refresh their math skills” (p. 231). Another study by Hawley and Chiang 
(2017) looked specifically at the effects of developmental education for adult 
undergraduate students at Ohio public community colleges and found more 
mixed results. While participation in developmental education was asso- 
ciated with some positive effects on short-term outcomes like persistence, 
there were statistically significant and negative effects on degree completion 
for adult students in developmental education. 


Developmental education reform and Florida’s SB 1720 


Over the past two decades, states and postsecondary education systems have 
responded to the challenges of low success rates in traditional developmental 
education programs by experimenting with various types of reforms. Early 
reform efforts in the 2000s tended to be small in scope and only provided 
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short-term support, while more recent efforts have struggled with challenges 
such as scaling up initiatives so that they are available to the majority of 
underprepared students (Jaggars & Bickerstaff, 2018). Florida’s developmental 
education reform under SB 1720 has been one of the most comprehensive, by 
requiring all FCS institutions to make three major changes system-wide 
relating to course placement, advising and support services, and the instruc- 
tional strategies used in developmental education courses. 

The first major change was that all students entering grade 9 after 2003/04 
and completing a standard high school diploma, as well as active-duty military 
personnel, became exempt from placement testing and developmental educa- 
tion courses. These students still had the option to take developmental educa- 
tion courses if they felt that they needed additional support, or they could 
enroll directly in the first college-level math course (Intermediate Algebra). 
There is very little evidence available about the outcomes of reforms that 
simply remove the requirement for developmental education, and the results 
are mixed. Kosiewicz and Ngo (2019) were able to take advantage of a natural 
experiment that occurred when a community college mistakenly allowed 
students to self-place into a developmental or college-level math course. 
They found that there were some improvements in student outcomes such 
as the likelihood of completing degree requirements in math, but the self- 
placement almost exclusively benefitted White, Asian, and male students. The 
authors suggest that these differences are likely due to women and students 
from underrepresented racial/ethnic groups being more likely to underesti- 
mate their abilities and opt into developmental education, and also a lack of 
adequate advising to help students make good decisions about placement. 

Florida’s reform differed in that it required all colleges to develop a plan to 
provide enhanced advising and academic support services for all incoming 
students, regardless of their initial course placement. Institutional leaders 
reported making advising changes such as adding more online orientation 
resources, increasing the duration of advising sessions, and spending addi- 
tional time with at-risk students identified through early alert systems (Hu 
et al., 2017). To provide additional academic support, colleges made changes 
such as adopting online tutoring programs, and adding or increasing faculty 
time in success centers or tutoring. These types of advising and academic 
support services have been shown to support student success in other contexts 
(e.g., Bettinger et al., 2013; Hatch & Garcia, 2017; Tampke, 2013). 

The final change under Florida’s SB 1720 was that colleges could no longer offer 
traditional semester-long developmental education courses and were instead 
required to offer developmental education courses using one or more instructional 
strategies including modularized, compressed, contextualized, and co-requisite. 
Modularized courses allow students to complete customized modules on content 
relating to only those skills that they have not yet mastered. Compressed courses 
contain the same amount of content as a semester-long developmental education 
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course, but they meet for a greater number of hours per week for fewer total weeks, 
potentially allowing students to complete two compressed courses in a single 
semester. Contextualized courses present developmental content in an applied 
manner related to students’ major course pathways, or meta-majors. Co-requisite 
courses allow students to take a developmental education course and a gateway 
course in the same subject area concurrently in the same semester. Prior research 
on these types of accelerated developmental education strategies has demonstrated 
positive impacts on short-term outcomes such as gateway course enrollment and 
completion rates, as well as longer-term outcomes including degree attainment 
(Denley, 2015; Hodara & Jaggars, 2014; Jaggars et al., 2014; Kalamkarian et al., 
2015; Okimoto & Heck, 2015). 

There is a growing body of evidence that Florida’s SB 1720 has yielded 
significant improvements in student success overall. Beginning in the first year 
of the reform in 2014, fewer students enrolled in developmental education, 
particularly in math, which declined from enrollment rates of 38% of incoming 
students in fall 2013 to 22% in fall 2014 (Hu et al., 2016). When examining the 
overall impact of the reform for both exempt and nonexempt students, the 
likelihood of enrolling in a college-level math course in the first year increased by 
8 percentage points and the likelihood of completing these courses increased by 
about 4 percentage points (Park-Gaghan et al., 2020). Moreover, the improve- 
ments in first-year coursetaking outcomes were greater for Black and Hispanic 
students relative to White students, thus reducing (and in some cases even 
eliminating) prior racial/ethnic achievement gaps. The impacts of the reform 
also differed by students’ high school preparation, with greater gains for students 
on the lowest two academic tracks (Park-Gaghan et al., 2019). A similar trend 
was seen in the impact of SB 1720 on credit accumulation, with significant gains 
in the number of credits attempted and earned in the first year, particularly for 
Black, Hispanic, low-income, and underprepared student (Mokher et al., in 
press). Taken together, these findings suggest that it is important to examine 
the heterogeneous impacts of SB1720 among student subgroups, as the reform 
seems to have benefitted some students more than others. 

The focus of the present study is the subgroup of nonexempt students who were 
still required to participate in placement testing and enroll in developmental 
education if they scored below college-ready. These nonexempt students made 
up about one-third of FTIC students, so a substantial number of students were 
affected by these nonexemption criteria. Focusing on nonexempt students allows 
us to look at the performance of students assigned to new accelerated instructional 
strategies for developmental education relative to those in traditional develop- 
mental education courses prior to the reform. We also examine how outcomes for 
students in these new accelerated strategies compare to similar students who 
narrowly avoided developmental education and could enroll directly into college- 
level courses with additional support services. 
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Data 


The data consist of student-level records for all first-time-in-college (FTIC) 
students at Florida’s 28 state colleges who were nonexempt from develop- 
mental education under SB 1720. Our data include college enrollment records 
and transcripts, as well as demographic characteristics. We also have high 
school records for students who attended a Florida public high school, but 
these records are missing for the majority of nonexempt students so we are 
unable to account for high school variables in our analyses. Table 1 describes 
the reasons that students were classified as nonexempt and the share of 
students classified as nonexempt for each reason. The largest group is students 
who attended a non-Florida public high school (52%), followed by students 
who attended a Florida public high school but did not complete a standard 
diploma (26%). Only about 5% of students were nonexempt because they 
entered grade 9 in a Florida public high school prior to 2003/04. Additionally, 
about 3% were nonexempt for multiple reasons—such as entering grade 9 
prior to 2003/04 and not completing a standard high school diploma. The 
reasons for exemption are unknown for the remaining 15% of students 
because their administrative records are incomplete. 

Our sample includes two cohorts of FTIC students prior to the reform (Fall 
2012 and 2013) and two cohorts post-reform (Fall 2014 and 2015). We limit 
the analytic sample to students who have scores on the college placement test, 
which is over 70% of nonexempt students in both the pre- and post-policy 
years. Students may be missing test scores due to incomplete administrative 
records, or if they have scores on alternate placement tests including the SAT, 
ACT, or Accuplacer (which are not observed in our data). Students missing 
PERT scores are less likely to be Black and more likely to be Hispanic 
(standardized mean differences of 0.28 and 0.27, respectively), and also tend 
to be about 2.3 years older (standardized mean difference of 0.27). 
Demographic characteristics for White, other race, and female are similar 
for students with missing and non-missing PERT scores. 

Each cohort has approximately 25,000 nonexempt students. For the post- 
reform years, the data includes an indicator for whether the student met the 
exemption criteria under SB 1720. In order to create a comparable variable for 
the pre-policy years, we classified students as nonexempt if they (a) did not 


Table 1. Number and percent of nonexempt students, by reason for 
nonexemption. 


Reason for nonexemption N Percent 
Non-FL public school 47,344 52.02 
FL public school, but no standard diploma 23,220 25.51 
FL public school, but in grade 9 prior to 03/04 4,571 5.02 
Multiple reasons 2,459 2.70 
Unknown 13,423 14.75 


Total 89,892 100.00 
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have a transcript record from a Florida public high school, (b) did not 
complete a standard high school diploma, or (c) completed a standard high 
school diploma prior to 2006/07. We do not have a valid indicator for active- 
duty military personnel status so we are unable to include this in the pre-policy 
exemption variable. However, this should have little impact on the sample 
given that less than 1% of students enrolled in FCS institutions are classified as 
active duty (Florida College System, 2018). 

The sequence of math courses available to students at FCS institutions 
consists of lower-level developmental education, upper-level developmental 
education, the first college-level math, and gateway math. While the specific 
courses taken may vary somewhat by institutions, we provide a description of 
a typical sequence based on information provided by Valencia College (n.d.). 
The lower-level course, Developmental Mathematics I (MAT 0018), is the first 
course in a college-preparatory two-course sequence emphasizing fundamental 
operations with applications to beginning algebra. The upper-level course, 
Developmental Mathematics II (MAT 0028), is the second course in the college- 
preparatory sequence which provides algebraic background on topics including 
fundamental operations with polynomials, linear equalities, factoring, and an 
introduction to radicals. The first college-level math course, Intermediate 
Algebra (MAT 1033) counts for elective credit only, and develops algebraic skills 
on topics such as systems of linear equations and inequalities, introduction to 
functions, complex numbers, and quadric equations.’ Next in the sequence is 
the first gateway math course, which fulfills the associate’s degree requirements 
in math. Students have the option to choose from among several different 
gateway courses including College Algebra (MAC1105), Elementary Statistics 
(STA2023), Liberal Arts Math I (MGFX106), and Liberal Arts Math II 
(MGEFX107). 

The outcomes of interest for this study include dichotomous indicators for 
whether students took and passed the first college-level math course 
(Intermediate Algebra), and a gateway math course. These outcomes are 
measured by the end of the first year of college enrollment. Control variables 
for student background characteristics include a series of dichotomous vari- 
ables for race (White, Black, Hispanic, other), a continuous variable for age 
(in years), and a dichotomous variable for sex (1 = female, 0 = male). There 
was no missing data for any of these variables. On average, nonexempt 
students tend to be similar to exempt students on all demographic charac- 
teristics except for age (Table 2). While nonexempt students tend to be about 
6 years older than exempt students (25.0 versus 19.3 years), it is important to 
note that many nonexempt students do not meet the criteria of 25 years or 
older typically used to define adult undergraduates. The racial/ethnic dis- 
tribution of nonexempt students is 40.6% White, 20.0% Black, 32.1% 
Hispanic, and 7.3% other. Approximately 53% of nonexempt students are 
female. 
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Table 2. Mean (and standard deviation) of student background characteristics, by exempt status 
and policy status. 


Exempt status Policy status (exempt only) 


Nonexempt Exempt Std. mean difference Pre-policy Post-Policy Std. mean difference 


Percent White 40.6 38.8 —0.04 40.9 40.3 -0.01 
40.9) 48.7 49.2) (49.1 

Percent Black 20.0 21.3 0.03 21.7 18.0 —0.09 
40.0) 40.9 (41.2) (38.5 

Percent Hispanic 32.1 33.6 0.03 30.7 33.8 0.07 
46.7) 47.3 46.1) (47.3 

Percent other race 7.3 6.00 —0.05 6.7 7.9 0.04 
26.0) 23.9 (25.1) (27.0 

Percent female 53.0 51.8 —0.02 52.8 53.3 0.01 
49.9) 50.0 49.9) (50.0 

Average age 25.0 19.3 —-0.91 26.2 23.7 —0.27 
(9.4) (3.1) (10.0) (8.6) 


Standardized (Std.) mean differences are calculated as the difference between each group divided by the pooled 
standard deviation. 


We also compared the demographic characteristics of the nonexempt 
students in the pre-policy years to the post-policy years. The standardized 
mean differences were less than 0.10 for all demographic characteristics except 
for age. The pre-policy cohort of exempt students was slightly older, with an 
average age of 26.2 years compared to 23.7 years in the post-policy cohort. 


Methods 


We examine the impact of assignment to developmental education course- 
work on enrollment and passing rates in first-year college courses in math for 
nonexempt students, and whether this relationship changes with the imple- 
mentation of Florida’s SB 1720. We use a regression discontinuity (RD) design 
since assignment to the treatment (in this case, developmental education) is 
determined by a continuous running variable (the Postsecondary Education 
Readiness Test, or PERT). Scaled scores on the PERT range from 50 to 150, 
with students assigned to lower developmental education for scores below 96, 
upper developmental education for scores between 96 and 112, and 
Intermediate Algebra for scores of 113 or above.” We model the likelihood 
of taking and passing various math courses in the first year as a function of 
whether an individual’s score on the PERT fell above or below the cutoff for 
lower-level developmental courses in the first set of models, and for the cutoff 
for upper-level development courses in the second set of models. 

Not all students who are assigned to developmental education courses 
proceed to take these courses. Many students delay enrollment in their first 
math course and may take no math in the first year. Additionally, some 
students may have concordance scores on other tests like ACT or SAT, 
which are not observed in our data. However, Figure 1 shows that there are 
sharp discontinuities in enrollment in developmental education courses near 
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Figure 1. Compliance with assignment to treatment based on the relationship between PERT 
scores (centered around the cutoff for assignment to developmental education) and enrollment in 
any developmental course (upper-level cutoff) and enrollment in a lower-level developmental 
course (lower-level cutoff). 
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the placement score cutoffs. At the upper developmental cutoff, less than 2% of 
students scoring above the cutoff take any developmental math course in the 
first year compared to nearly 70% of students below the cutoff. At the lower 
developmental cutoff, less than 2% of students scoring above the cutoff take 
a lower developmental math course compared to 50% of students below the 
cutoff. Since this is a sharp RD analysis, we are estimating the intent-to-treat 
effect based on assignment to developmental education courses. Therefore, 
this estimate represents the overall impact of the reform for all nonexempt 
students, regardless of whether or not they complied with an assignment to 
their treatment condition (placement into developmental education or col- 
lege-level courses). This estimate may be particularly relevant from a policy 
perspective, as it represents the impacts of the reform under actual implemen- 
tation rather than under ideal conditions. The primary advantage of RD is that 
it provides a rigorous estimate of the program impact, because any differences 
in student outcomes should be attributable to the treatment if assignment to 
the treatment is the only thing that changes at the cutoff. However, one 
limitation of this approach is that the treatment effects are only estimated 
for students near the cutoff, and may not be generalizable to the full sample of 
nonexempt students. 

Using 2 years of data before and after the policy, we employ a Difference-in- 
Regression Discontinuity (DiRD) to empirically determine whether the RD 
estimates change following the policy implementation (Murnane & Willett, 
2011). This model allows us to combine pre- and post-policy cohorts to test 
whether the RD estimates for assignment to developmental education change 
after the reform. Following the approach of Cattaneo et al. (2017, 2018), we 
identify optimal bandwidths by minimizing the mean squared error (MSE) of 
the local polynomial RD point estimator, while allowing the left and right 
bandwidths to differ. We estimate the following logit model for student i at 
college j in cohort (year) t: 


Logit(yit) = « + B(DE 1(POST,) + ¢(DE;j * POST) + p(PERT i+) 
9(S) 1) 4 + A; Eijt 

where y is the outcome of interest (e.g., enrolling in a college-level math 
course), DE is an indicator of whether the student was assigned to DE based on 
the placement test score (1 = yes, 0 = no), POST is an indicator for the post- 
policy cohorts (1 = yes, 0 = no), and DE*POST is an interaction between the 
two. We also control for the placement test score (PERT) centered at the cut 
score, measures of student background characteristics (S), year fixed effects 
(A,), and college fixed effects (y;) to control for unobserved heterogeneity 
across institutions, with standard errors clustered at the institutional level. 
For ease of interpretation, we also present predicted probabilities from the 
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regression models for students above and below the cutoffs in the pre-reform 
and post-reform periods. 

The DiRD estimates for the DE coefficient address our first research ques- 
tion of whether first-year coursetaking outcomes differ based on assignment to 
developmental education for nonexempt students. Based on numerous prior 
studies of developmental education, we expect that the likelihood of taking 
and passing college-level courses in the first year would be lower for students 
scoring below the college-level cut score who are placed into developmental 
education, relative to similar students scoring just above this cut score who 
narrowly avoid developmental education. The estimates for the POST coefh- 
cient address our second research question about the effect of the develop- 
mental education reform on first-year coursetaking outcomes for nonexempt 
students. For example, a statistically significant positive coefficient on POST 
would indicate that students tend to have a greater likelihood of taking or 
passing college-level courses in the post-reform years relative to the pre- 
reform years. 

The coefficient for DE*POST addresses our third research question about 
whether the effect of the reform differs depending on nonexempt students’ 
assignment to developmental education. This is important because it allows us 
to examine whether the new instructional strategies—coupled with enhanced 
support services—tend to be more effective than traditional developmental 
education courses offered in the pre-reform period. While prior studies on 
Florida’s reform have focused on what happens when developmental educa- 
tion is made optional for the majority of students, this study is the first to 
examine what happens to the nearly one-third of remaining students who 
could not opt out. The results provide context for understanding whether the 
benefits of Florida’s reform may be attributed solely to removing mandatory 
assignment to developmental education for exempt students, or whether there 
may also be benefits to changing the ways in which developmental education is 
offered for nonexempt students. 

Another benefit of these analyses is that we can examine whether the effects 
of assignment to developmental education before and after the reform change 
differentially depending on students’ level of preparation. For example, we 
might anticipate that students who are close to college-ready may perform well 
in an accelerated developmental course accompanied by additional support 
services. But would students who are far from college-ready also benefit, or 
would they be better off in a traditional semester-long course? We run separate 
DiRD models for two different PERT cutoffs, in which each has a different 
sample. The first cutoff is for students who are assigned to a lower-level 
developmental education course. Prior to the reform, these students would 
have been required to complete one lower-level developmental course and one 
upper-level developmental course before enrolling in the first college-level 
course. Given that these courses were designed to be taken in a sequence, 
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this means that most students would not be ready to enter a college-level math 
course until at least their third semester of college. However, after the reform 
many colleges combined both lower- and upper-level developmental educa- 
tion in a single course with additional credit hours, so that students could 
potentially fulfill the developmental education requirements within one seme- 
ster. The second cutoff is for students who are assigned to an upper-level 
developmental education course. These students would have been assigned to 
a traditional semester-long developmental education course prior to the 
reform. After the reform, they would have been assigned to take 
a developmental course using a new instructional strategy, such as 
a compressed or co-requisite course, which would allow them to complete 
the developmental education requirements concurrently with the first college- 
level math course in the same semester. By including a separate set of models 
for both the upper and lower developmental cutoffs in our analyses, we are 
able to examine whether the effects of the reform may differ for students with 
different levels of academic preparation. 

We also conduct several sensitivity tests to assess the robustness of our results. 
First, we estimate a baseline model with no covariates to determine whether the 
results differ depending on whether we control for other factors including 
student demographic characteristics and college fixed effects. Second, we esti- 
mate our models using a variety of different functional forms for the relationship 
between the dependent variable and the running variable, including quadratic, 
cubic, and quartic specifications. If the underlying relationship is incorrectly 
specified then the treatment effect may be biased, as the functional form can 
create an apparent shift that suggests a discontinuity when there is none (Jacob 
et al., 2012). Third, we assess the sensitivity of the results in the bandwidth 
specification by using coverage error rate (CER) optimal bandwidths in addition 
to the MSE optimal bandwidths in our preferred specification (Cattaneo et al., 
2017, 2018). Fourth, we estimate models including interaction terms between 
developmental education assignment and PERT scores (DE*PERT), as well as 
developmental education assignment and PERT scores and the post-policy 
indicator (DE*PERT*POST) to determine whether the treatment may impact 
the slope of the regression line. Overall, we find that our results are highly robust 
across all of these sensitivity tests, as shown in the Appendix Tables A1-A8. 


Validity of RD estimates 


We begin by assessing whether our analyses satisfy the four standards for 
sharp RD studies using the criteria specified by the What Works 
Clearinghouse (What Works Clearinghouse, 2017). The first standard is to 
satisfy the integrity of the running variable, which means demonstrating that 
students’ scores have not been manipulated. The PERT has institutional 
integrity because it is scored by an external provider (McCann Associates) 
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with no incentive to modify students’ placement status. Additionally, if the 
student took the PERT more than once, we use the first test score since this 
cannot be manipulated due to selective retesting. We further test the statistical 
integrity of the running variable by conducting an RD manipulation test using 
local polynomial density estimation (Cattaneo et al., 2017). For both cutoffs, 
we fail to reject the null hypothesis of no difference in the density of the treated 
and control observations (p = .255 for the upper-cutoff and p = .206 for the 
lower-cutoff). These findings are further supported by a graphical analysis of 
the distribution of PERT scores, which shows no strong evidence of any 
discontinuities in students’ test scores (Figure 2). 

The second standard is to ensure low levels of attrition, both overall and by 
treatment status. We have outcome data for all students, so no observations are lost 
due to attrition. If students drop out of college before taking or completing 
a college-level math course, then they are assigned a value of 0 for the dichotomous 
outcomes indicating whether students took and passed college-level math courses. 

The third standard is to satisfy the continuity of the relationship between the 
outcomes and the running variable. The first criterion within this standard is to 
demonstrate baseline equivalence on key covariates for students above and below 
the cutoffs. In our sample, all values have a standardized mean difference of less 
than 0.15 standard deviation units, which is well below the WWC maximum of 
0.25 (Table 3). We also control for the full set of covariates in our models since 
some standardized mean differences were greater than 0.05. The second criterion 
is to provide graphical evidence that there are no discontinuities in the running 
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Figure 2. Density plot of the running variable, PERT math scores. 
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Table 3. Baseline equivalence on student background characteristics for students above and below 
the cutoffs for assignment to lower-level developmental education and upper-level developmen- 
tal education. 


Below cutoff Above cutoff 
Mean Std. Dev. Mean Std. Dev. Std. mean difference 
Assignment to lower-level developmental courses 
White 0.46 0.50 0.43 0.49 0.06 
Black 0.24 0.43 0.19 0.39 0.13 
Hispanic 0.27 0.45 0.28 0.45 —0.03 
Other race 0.06 0.23 0.07 0.25 —0.04 
Female 0.50 0.50 0.53 0.50 —0.06 
Age 23.94 771 23.21 7.25 0.10 
Cohort 2013.38 1.09 2013.54 1.08 —0.14 
Assignment to upper-level developmental courses 
White 0.43 0.49 0.42 0.49 0.01 
Black 0.15 0.36 0.14 0.34 0.05 
Hispanic 0.34 0.47 0.33 0.47 0.00 
Other race 0.09 0.28 0.10 0.30 —0.04 
Age 21.25 5.78 21.22 5.91 0.01 
Female 0.50 0.50 0.52 0.50 —0.04 
Cohort 2013.61 1.11 2013.72 1.10 —0.10 


Standardized (Std.) mean differences are calculated as the difference between each group divided by the pooled 
standard deviation. 


a b 


Enrollment in college-level math 
Completion of college-level math 


o , 


. 
. 
. ? eee 
° 
o4 G : T T 


= 
0 2 60 80 100 120 140 
60 80 100 120 140 

PERT math score PERT math score 


Enrollment in gateway math 
Enrollment in gateway math 


. . 
. ° 
* eo 


» 
oe oeeet? 


” 
ale eo © oe ee® 


60 80 120 140 60 80 


100 100 120 140 
PERT math score PERT math score 


Figure 3. Scatterplots of the relationship between the outcome variables and the running variable 
(PERT math scores). Solid lines indicate the cut scores for assignment to lower- and upper-level 
developmental education. 


variable at any values other than the cutoff. Figure 3 provides scatterplots of the 
outcomes and running variables, and the only discontinuities are at the scores for 
assignment to upper- and lower-level developmental courses. 
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The fourth standard is to satisfy the functional form and bandwidth selections. 
The first criterion is that the statistical model must control for the running variable, 
which is done in all of our models. The second criterion is that a local regression or 
related nonparametric approach is used with a justified bandwidth. Our regression 
models are estimated using MSE-optimal bandwidths, and sensitivity tests using 
CER-optimal bandwidths yield similar results. Given that we have a large sample 
that pools across multiple cohorts, our bandwidths are relatively small, ranging 
from 3 to 6 points above or below the cut score (on a scale of 50 to 150) depending 
on the model. The next criterion is evidence that the findings are robust to varying 
bandwidths and functional forms. In the appendix (Tables A1l-A8), we demon- 
strate that our results are robust to the choice of bandwidth (MSE or CER), to the 
choice of functional form (linear, quadratic, cubic, and quartic), and to the 
inclusion of interaction terms between the treatment variable and the forcing 
variable. The next criterion is a graphical analysis displaying the relationship 
between the outcome and the running variable. The graph must demonstrate 
consistency with the choice of bandwidth and the functional form specification. As 
shown in Figure 3, the relationship appears to approximate a linear trend, although 
we do conduct sensitivity tests using other functional forms. The last criterion is 
that the relationship between the running variable and the outcome must not be 
constrained to be the same on both sides of the cutoff. Our optimal bandwidths 
allow for variation on the left and right of the cutoff. 


Findings 
Impact of assignment to upper-level developmental education 


We begin by examining differences in first-year coursetaking outcomes in math 
among students narrowly assigned into a single developmental education math 
course (upper DE) relative to those who are narrowly assigned to the first 
college-level course (Intermediate Algebra). As a reminder, these are local 
average treatment effects and are not necessarily generalizable to the whole 
population of nonexempt students. The results for the DiRD models are pre- 
sented in Table 4, with predicted probabilities and marginal effects in Table 5. 
The statistically significant negative coefficients on the upper DE variable indi- 
cate that students assigned to upper DE were less likely to enroll in and complete 
both Intermediate Algebra (the first college-level math course) and gateway 
math courses (the next college-level math course in the sequence) relative to 
students who were directly placed into Intermediate Algebra. For example, the 
predicted probability of enrolling in Intermediate Algebra prior to the reform 
was 60.1% for students assigned to upper DE versus 72.8% for students directly 
assigned to this course. Enrollment in gateway courses was much lower for 
students on both sides of the cutoff, with predicted probabilities of 20.6% for 
students assigned to upper DE versus 31.2% for students assigned to 
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Table 4. Impact of assignment to upper DE (relative to college-level) on first-year coursetaking 
outcomes for nonexempt students. 


(1) Enrollment in (2) Completion of (3) Enrollment in (4) Completion of 
Intermediate Algebra Intermediate Algebra gateway math gateway math 
(or higher) (or higher) (or higher) (or higher) 
Upper DE —0.572*** —0.487*** —0.558*** —0.579*** 
(0.107) (0.092) (0.104) (0.116) 
Post 0.383*** 0.320** 0.532*** 0.515*** 
(0.104) (0.113) (0.105) (0.100) 
Upper DExPost —0.103 —0.069 0.407*** 0.348*** 
(0.093) (0.086) (0.084) (0.084) 
Bandwidth [-6, +5] [-6, +5] [-6, +5] [-6, +5] 
N 8,968 8,968 8,968 8,968 


Notes. Results are from a Difference-in-Regression Discontinuity model using optimal bandwidths by minimizing the 
mean squared error (MSE) of the local polynomial RD point estimator. Models include controls for the placement 
test score centered at the cut score, measures of student background characteristics, year fixed effects, and college 
fixed effects to control for unobserved heterogeneity across institutions, with standard errors clustered at the 
institutional level. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10. 


Table 5. Predicted probabilities for first-year math coursetaking outcomes for students assigned to 
upper DE relative to college-level math. 


Pre-Reform Post-Reform Difference 

Enrollment in Intermediate Algebra (or higher) 

Upper DE 60.1% 66.6% 6.5%*** 

College-level 72.8% 79.6% 6.9%*** 
Marginal Effects 

Upper DE vs. college-level 0.4% 
Completion of Intermediate Algebra (or higher) 

Upper DE 43.6% 49.8% 6.2%*** 

College-level 55.6% 63.2% 7.6%*** 
Marginal Effects 

Upper DE vs. college-level -14% 
Enrollment in gateway math (or higher) 

Upper DE 20.6% 39.9% 19.3%*** 

College-level 31.2% 43.6% 12.4%*** 
Marginal Effects 

Upper DE vs. college-level 6.9%*** 
Completion of gateway math (or higher) 

Upper DE 14.1% 28.0% 13.9%*** 

College-level 22.6% 32.8% 10.2%*** 
Marginal Effects 

Upper DE vs. college-level 3.7%*** 


Results are predicted probabilities from a Difference-in-Regression Discontinuity model using optimal bandwidths by 
minimizing the mean squared error (MSE) of the local polynomial RD point estimator. Models include controls for 
the placement test score centered at the cut score, measures of student background characteristics, year fixed 
effects, and college fixed effects to control for unobserved heterogeneity across institutions, with standard errors 
clustered at the institutional level. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10. 


Intermediate Algebra in the pre-reform period. These findings suggest that in 
response to our first research question about how outcomes differ based on 
assignment to developmental education, nonexempt students assigned to upper 
DE tend to perform worse on first-year coursetaking outcomes compared to 
similar students not assigned to developmental education courses. 

In response to our second research question concerning the effects of the 
reform, we find that the developmental education reform tends to have 
a positive effect on first-year coursetaking outcomes for nonexempt students. 
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The statistically significant positive coefficients on the POST variable indicate 
that the reform is associated with improvements on all four outcomes (enroll- 
ment in Intermediate Algebra, completion of Intermediate Algebra, enroll- 
ment in gateway math, and completion of gateway math) for all students near 
the margins of college-readiness. In Intermediate Algebra, both enrollment 
and passing rates increased by about 7 percentage points after the reform for 
students both above and below the upper DE cutoff. 

For research question three about whether there are differential effects of 
the reform, we find some evidence that the reform had greater effects for 
students assigned to upper DE. On the outcomes of enrollment and comple- 
tion of gateway math courses, the statistically significant and positive coefh- 
cients on the UpperDE X POST interaction term indicates that students who 
were assigned to upper DE experienced larger gains under the reform relative 
to students assigned to Intermediate Algebra. In other words, the new devel- 
opmental education approach as stipulated in SB 1720 has better results as 
measured by student first-year course outcomes than the traditional approach 
before the reform. The likelihood of enrolling in a gateway math course 
increased by 19.3 percentage points for students assigned to upper DE, relative 
to an increase of 12.4 percentage points for students assigned to Intermediate 
Algebra (a difference of 6.9 percentage points). The likelihood of passing 
a gateway math course increased by 13.9 percentage points for students 
assigned to upper DE, relative to an increase of 10.2 percentage points for 
students assigned to Intermediate Algebra (a difference of 3.7 percentage 
points). This indicates that the gap in performance on gateway outcomes by 
developmental education placement status narrowed after the reform. 


Impact of assignment to lower-level developmental education 


The second set of results in Tables 6 and 7 examines the same first-year math 
coursetaking outcomes for students narrowly assigned to two developmental 
math courses (lower DE) relative to students narrowly assigned to a single 
developmental math course (upper DE). In response to the first research 
question about how outcomes differ based on assignment to developmental 
education, there is some evidence that first-year coursetaking outcomes are 
worse for students assigned to lower DE. The statistically significant negative 
coefficients on the lower DE variable in the first two models indicate that 
students below the cutoff who were assigned to lower DE were significantly 
less likely to enroll in or complete Intermediate Algebra (the first college-level 
math course) relative to students who were assigned to upper DE. Prior to the 
reform, the predicted probabilities of enrolling in Intermediate Algebra were 
15.8% for students assigned to lower DE and 33.3% for students assigned to 
upper DE, and the predicted probabilities of passing this course were 8.4% for 
lower DE students and 17.5% for upper DE students. The coefficients on the 
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Table 6. Impact of assignment to lower DE (relative to upper DE) on first-year coursetaking 


outcomes for nonexempt students. 


Enrollment in 


Intermediate Algebra 


Completion of 
Intermediate Algebra 


Enrollment in gate- Completion of gate- 


way math (or 


way math (or 


(or higher) (or higher) higher) higher) 
Lower DE —0.971*** —0.841*** —0.253 —0.375 
(0.149) (0.146) (0.213) (0.293) 
Post 0.107 0.433*** 0.898*** 0.948*** 
(0.129) (0.109) (0.204) (0.272) 
LowerXPost 0.722*** 0.606*** 0.200 0.173 
(0.115) (0.134) (0.175) (0.183) 
Bandwidth [-4, +6] [-4, +6] [-4, +6] [-4, +6] 
N 13,772 13,772 13,772 13,772 


Notes. Results are from a Difference-in-Regression Discontinuity model using optimal bandwidths by minimizing the 
mean squared error (MSE) of the local polynomial RD point estimator. Models include controls for the placement 
test score centered at the cut score, measures of student background characteristics, year fixed effects, and college 
fixed effects to control for unobserved heterogeneity across institutions, with standard errors clustered at the 
institutional level***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10. 


Table 7. Predicted probabilities for first-year math coursetaking outcomes for students assigned 
to lower DE relative to upper DE math. 


Pre-Reform Post-Reform Difference 

Enrollment in Intermediate Algebra (or higher) 

Lower DE 15.8% 30.2% 14.3% 

Upper DE 33.2% 35.7% 2.5% 
Marginal Effects 

Lower DE vs. Upper DE 11.8%*** 
Completion of Intermediate Algebra (or higher) 

Lower DE 8.4% 20.6% 12.2%*** 

Upper DE 17.5% 24.7% 7.2%*** 
Marginal Effects 

Lower DE vs. Upper DE 5.0%*** 
Enrollment in gateway math (or higher) 

Lower DE 2.7% 7.7% 5.0%*** 

Upper DE 3.4% 8.0% 4.6%*** 
Marginal Effects 

Lower DE vs. Upper DE 0.4% 
Completion of gateway math (or higher) 

Lower DE 1.5% 4.5% 3.0%*** 

Upper DE 2.2% 5.4% 3.3%" 
Marginal Effects 

Lower DE vs. Upper DE —0.2% 


Results are predicted probabilities from a Difference-in-Regression Discontinuity model using optimal bandwidths 
by minimizing the mean squared error (MSE) of the local polynomial RD point estimator. Models include controls 
for the placement test score centered at the cut score, measures of student background characteristics, year fixed 
effects, and college fixed effects to control for unobserved heterogeneity across institutions, with standard errors 
clustered at the institutional level. ***p < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10. 


lower DE variable were not significant in models 3 and 4, indicating there were 
no statistically significant differences by placement status in the likelihood of 
taking and passing gateway math courses, with very low predicted probabil- 
ities of 3% or less on these outcomes for both groups of students prior to the 
reform. These low enrollment and passing rates are not surprising, given that 
gateway math would have been the fourth course in the math sequence under 
a traditional developmental program (lower DE, upper DE, Intermediate 
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Algebra, and gateway math) so it would have been very difficult for lower- 
performing students to progress this far in their first year. 

Our second research question examines whether the reform has an effect on 
first-year coursetaking outcomes for nonexempt students assigned to lower 
DE, and as with the results for the upper DE cutoff, we find some evidence that 
the reform did result in gains in student outcomes. The statistically significant 
positive coefficients on the POST variable in models 2, 3, and 4 indicate the 
reform was associated with an increase in three of the four coursetaking 
outcomes (completion of Intermediate Algebra, enrollment in gateway math, 
and completion of gateway math). 

The third research question addresses whether the effects of the reform 
differ depending on nonexempts students’ assignment to developmental edu- 
cation. The statistically significant and positive coefficients on the LowerDE 
X POST interaction term in models 1 and 2 indicate that the effects of the 
reform on the Intermediate Algebra outcomes were greater for students’ 
assigned to lower DE. The predicted probability of enrolling in Intermediate 
Algebra increased after the reform by 14.3 percentage points for students 
assigned to lower DE, relative to only 2.5 percentage points for students 
assigned to upper DE (a difference of 11.8 percentage points). Additionally, 
the predicted probability of completing Intermediate Algebra after the reform 
increased by 12.2 percentage points for students assigned to lower DE relative 
to 7.2 percentage points for students assigned to upper DE. This indicates that 
the gap in performance on Intermediate Algebra outcomes by developmental 
education placement status narrowed after the reform. There were no statis- 
tically significant effects of the LowerDE X POST interaction terms in models 
3 and 4, which suggests that the effects of the reform were similar by DE 
placement status for the gateway coursetaking outcomes. For students on both 
sides of the placement cutoff, the predicted probabilities increased after the 
reform for enrollment in gateway math by about 5 percentage points and for 
completion of gateway math by about 3 percentage points. 


Discussion 


While several states have experimented with developmental education 
reforms such as using multiple measures for placement decisions, Florida’s 
reform went a step further by letting most students decide whether to opt out 
of developmental education courses. However, there were concerns that 
some students who had not recently graduated with a standard diploma 
from a Florida public high school may not be adequately prepared for 
college-level courses, and these students were not exempt from prior require- 
ments for placement testing and developmental education (for those scoring 
below college-ready). This study provides additional insight into the policy 
decision to leave this group of students subject to these traditional 
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requirements, and has important implications for policymakers in Florida 
and other states as they consider who may benefit from being allowed to opt- 
out of developmental education. This is a policy issue of increasing rele- 
vance, as California has recently eliminated all free-standing developmental 
education courses in the California State University system, and is moving 
toward doing the same in the community college system beginning in the 
2020/21 academic year (Mangan, 2019). There are concerns that this 
approach could harm some students in the absence of greater efforts to 
improve the academic preparation provided in K-12 schools to incoming 
students (e.g., Kurlaender, 2018). 

Similar to other RD studies examining the effects of placement into devel- 
opmental education (e.g., Valentine et al., 2017), we find that first-year math 
coursetaking outcomes tend to be worse for nonexempt students assigned to 
additional developmental education requirements relative to similar students 
scoring just above college-ready. This suggests that more nonexempt students 
may have benefitted from the option to enroll directly into college-level 
courses like their exempt peers, although the results may not be generalizable 
to all nonexempt students. Policymakers may want to consider removing 
exemption criteria and providing all students with the choice of opting out 
of developmental education. This finding is also consistent with results from 
the larger study of the impact of Florida’s developmental education reform for 
all students (Hu et al., 2019; Park-Gaghan et al., 2020), which finds that 
removing developmental education requirements tends to be beneficial for 
student outcomes when additional support services are provided. 

We also find that the reform has positive effects on first-year coursetaking 
outcomes for students above and below both cutoffs for assignment to devel- 
opmental education in math. The reform is beneficial for students just above 
the developmental cutoffs who are assigned to college-level courses and have 
additional supports, but overall the effects tend to be greater for students who 
have both additional support plus accelerated instructional strategies for 
developmental education. These gains were often quite large in magnitude, 
particularly for a statewide reform. This is consistent with the majority of prior 
research examining differential effects of developmental education by stu- 
dents’ academic preparation, which have found that lower-performing stu- 
dents who are assigned to multiple levels of developmental education tend to 
be less likely to progress to the next course in the sequence relative to students 
with fewer developmental requirements (Dadger, 2012; Melguizo et al., 2016; 
Ngo & Kosiewicz, 2017). 

For the lower DE cutoff, the reform has greater effects on taking and 
completing Intermediate Algebra (the first college-level course) for students 
who would have been assigned to two levels of developmental education 
relative to those assigned to one level of developmental education. For the 
upper DE cutoff, the reform has greater effects on taking and completing 
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gateway math courses (the next course in the math sequence) for students who 
would have been assigned to upper-level DE relative to those assigned to 
college-level math. These post-reform gains have narrowed preexisting gaps 
in coursetaking outcomes by developmental education placement status. Pre- 
post differences in outcomes for students placed into developmental education 
also suggest that providing students with developmental education courses 
through accelerated strategies with additional support services is more effec- 
tive than assigning students to traditional developmental education courses. 
Despite these gains, we still do not know whether some students may benefit 
from enrolling in optional developmental education courses, or if it is more 
effective to eliminate developmental education completely. At least one prior 
study (Boatman & Long, 2018) has found that some lower-performing stu- 
dents benefit from assignment to multiple levels of developmental courses. If 
policymakers decide to continue offering developmental education courses, 
they should consider alternate instructional strategies, such as those used in 
Florida, to help students progress through developmental courses more 
quickly. Other states like Texas have taken a similar approach by implement- 
ing legislation effectively mandating that co-requisites be the primary devel- 
opmental education model at all public institutions. 

Further, while it is difficult (if not impossible) for us to attribute gains in 
student outcomes to any single component of the Florida reform, we argue 
that Florida’s reform demonstrates the importance of thinking about how to 
package complementary reform efforts together synergistically. These types 
of comprehensive reform efforts may have a greater effect than if each 
component was implemented on its own as an isolated practice. 
Particularly for students at the lower DE cutoff, it may be important to 
provide additional support services so that students have a greater chance of 
succeeding in more accelerated course formats. Colleges were also given 
some flexibility in designing their plans about how to implement changes to 
developmental education courses and advising services. This aspect of 
Florida’s reform recognizes that what works in one setting with a specific 
population of students, may not be as effective in a different context. 
Allowing colleges to modify implementation based on their own unique 
contexts also likely contributed to the overall success of Florida’s reform. 
Policymakers in other states should also consider how to offer a package of 
complementary reform initiatives in a way that supports local adaptations to 
meet the needs of unique student populations. 

There were also some smaller gains after the reform for nonexempt 
students scoring above college-ready who were assigned to college-level 
classes. Even though most of these students did not participate in the 
accelerated instructional strategies for developmental education, they may 
have benefitted from the additional advising and academic support services 
that were available to all incoming students. Further, there is some evidence 
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that Florida’s developmental education reform led to institutional transfor- 
mation, which resulted in extensive changes to operational procedures, and 
cultural changes in the values and beliefs of diverse stakeholders throughout 
FCS institutions (Mokher et al., 2020). Even though the legislation did not 
require institutions to make changes to college-level and gateway courses, 
many institutional leaders reported making changes to instructional prac- 
tices and curriculum in these courses as part of a larger effort to improve 
student success in the midst of the developmental education reform. 

Both before and after the reform, enrollment rates in college-level and 
gateway math courses tend to be higher than the subsequent completion 
rates in these courses, which suggests that despite overall improvements in 
these outcomes post-reform, there are still some students who may not be 
adequately prepared to succeed in these courses. Future research could further 
explore whether there are certain subgroups of students who are more likely to 
be successful by beginning in an accelerated developmental course rather than 
enrolling directly into a college-level work. For example, prior research has 
found that older returning adults may be more in need of a “refresher” of math 
skills from high school (Bailey et al., 2010; Bettinger et al., 2013; Calcagno 
et al., 2007). Additionally, interviews with students themselves may provide 
insights into the extent to which students in accelerated developmental courses 
found these courses to be helpful. This type of information would also be 
useful for advisors who are tasked with helping students to make wise choices 
about course placement, often in the absence of test score information about 
students’ level of academic preparation. While the overall positive effects of 
Florida’s developmental education reform are encouraging, it is important to 
ensure, both in Florida and elsewhere, that there are no students outside of 
a given reform’s limelight who might be inadvertently left behind. 


Notes 


1. In some states, Intermediate Algebra is considered to be a developmental education 
course. However, in Florida, it is the first college-level math course and is a pre-requisite 
for almost all gateway math courses. SB 1720 only removed the requirement for devel- 
opmental education courses for exempt students, and all students were still required to 
take Intermediate Algebra unless they had placement test scores indicating that they 
were eligible to enroll in a gateway math course. 

2. In 2014 the cutoff for Intermediate Algebra increased by 1 point to 114. For the 2014 and 
2015 cohorts, students with a score of 113 are categorized as assigned to upper-level 
developmental courses. 
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Appendix DiRD sensitivity tests with alternate bandwidths, functional 
forms, and interaction terms 


Table A1. Sensitivity analyses with alternate bandwidths and functional forms for impact of 
assignment to upper DE (relative to college-level) on enrollment and completion of 
Intermediate Algebra (or higher) for nonexempt students. 


(1) Preferred specification w/MSE (2) No (3) (5) (6) CER-optimal 
optimal bandwidths covariates Quadratic (4) Cubic Quartic bandwidths 
Enrollment in Intermediate Algebra 
Upper DE —0.572*** 0.520*** —0.833*** -—0.814*** -—0.805*** —0.659** 
(0.107) (0.110) 0.141) (0.189) (0.214) (0.254) 
Post 0.383*** 0.460*** 0.380*** 0.380***  0.380*** 0.280 
(0.104) (0.104) 0.103) (0.103) (0.102) (0.174) 
Upper DE —0.103 0.129 0.100 0.100 0.099 —0.168 
X Post 
(0.093) (0.087) 0.092) (0.092) (0.093) 0.130 
Completion of Intermediate Algebra 
Upper DE —0.487*** —0.467*** = 0.543*** —0.734*** —0.726*** 0.563** 
(0.092) (0.093) 0.057) (0.137) (0.174) (0.111) 
Post 0.320** 0.359** 1.374** 0.318** 0.318** 1.299 
(0.113) (0.108) 0.155) (0.113) (0.113) (0.213) 
Upper DE —0.069 —0.080 0.935 —0.068 —0.068 1.043 
X Post 
(0.086) (0.082) 0.080) (0.086) (0.086) (0.126) 
Bandwidth [-6, +5] [-6,+5]  [-6,+5]  [-6,+5]  [-6, +5] [-3, +3] 
N 8,968 8,968 8,968 8,968 8,968 3,912 


Results are from Difference-in-Regression Discontinuity models with alternate bandwidths and functional forms. 
***D < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10. 


Table A2. Sensitivity analyses with alternate bandwidths and functional forms for the impact of 
assignment to upper DE (relative to college-level) on enrollment and completion of gateway math 


(or higher) for nonexempt students. 


(1) Preferred specification w/MSE (2) No (3) (5) (6) CER-optimal 
optimal bandwidths covariates Quadratic (4) Cubic Quartic bandwidths 
Enrollment in Gateway Math 
Upper DE -0.558*** -0.514*** — -0.636*** -0.589*** -0.606** -0.602** 
(0.104) (0.107) (0.113) (0.155) (0.190) (0.220) 
Post 0.532*** 0.576*** = 0.528*** = 0.528*** 0.528*** 0.643** 
(0.105) (0.110) (0.106) (0.106) (0.106) (0.114) 
Upper DE 0.407*** 0.369*** = -0.413*** = 0.414***  0.414*** 0.402** 
X Post 
(0.084) (0.080) (0.083) (0.084) (0.084) (0.137) 
Completion of Gateway Math 
Upper DE -0.579*** -0.549*** -0.624*** -0.582** -0.646** -0.615* 
(0.116) (0.115) (0.119) (0.207) (0.236) (0.276) 
Post 0.515*** 0.515*** = 0.512*** = 0.512*** § 0.512*** 0.734*** 
(0.100) (0.113) (0.100) (0.100) (0.100) (0.164) 
Upper DE 0.348*** 0.326*** = 0.352*** = 0.352*** = 0.351*** 0.300 
X Post 
(0.084) (0.083) (0.084) (0.084) (0.084) (0.167) 
Bandwidth [-6, +5] [-6, +5] [-6,+5]  [-6,+5]  [+6, +5] [-3, +3] 
N 8,968 8,968 8,968 8,968 8,968 3,912 


Results are from Difference-in-Regression Discontinuity models with alternate bandwidths and functional forms. 
***D < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10. 
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Table A3. Sensitivity analyses with alternate bandwidths and functional forms for the impact of 
assignment to lower DE (relative to upper DE) on enrollment and completion of Intermediate 
Algebra (or higher) for nonexempt students. 


(1) Preferred specification w/MSE (2) No (3) (5) (6) CER-optimal 
optimal bandwidths covariates Quadratic (4) Cubic Quartic bandwidths 
Enrollment in Intermediate Algebra 
Lower DE -0.971*** -0.945*** = -0.884*** -0.890*** -0.785*** -0.951*** 
(0.149) (0.152) (0.165 (0.157 (0.219) (0.196) 
Post 0.107 0.083 0.108 0.108 0.109 0.205 
(0.129) (0.138) (0.129 (0.129 (0.129) (0.183) 
Lower DE 0.722*** 0.707*** = 0.720*** = =0.720*** 0.720*** 0.583** 
X Post 
(0.115) (0.116) (0.115 (0.115 (0.115) (0.167) 
Completion of Intermediate Algebra 
Lower DE -0.841*** -0.832***  -0.637*** -0.620*** -0.347 -0.426* 
(0.146) (0.145) (0.161 (0.138 (0.221) (0.173) 
Post 0.433*** 0.396*** = 0.435*** = 0.435*** = 0.436*** 0.3347 
(0.109) (0.117) (0.108 (0.108 (0.108) (0.189) 
Lower DE 0.606*** 0.591*** 0.600*** 0.600*** 0.601*** 0.434* 
X Post 
(0.134) (0.133) (0.134 (0.135 (0.134) (0.192) 
Bandwidth [-4, +6] [-4, +6] [-4,+6]  [-4,+6]  [-4, +6] [-2, +3] 
N 13,772 13,772 13,772 13,772 13,772 5,073 


Results are from Difference-in-Regression Discontinuity models with alternate bandwidths and functional forms. 


***D < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10. 


Table A4. Sensitivity analyses with alternate bandwidths and functional forms for the impact of 
assignment to lower DE (relative to upper DE) on enrollment and completion of gateway math (or 
higher) for nonexempt students. 


Lower DE 


Post 


Lower DE 


X Post 


Lower DE 


Post 


Lower DE 
X Post 


Bandwidth 
N 


(1) Preferred specification w/MSE 


optimal bandwidths 


-0.253 
(0.213) 
0.898*** 
(0.204) 
0.200 


(0.175) 


-0.375 
(0.293) 
0.948*** 
(0.272) 
0.173 


(0.183) 
[-4, +6] 
13,772 


(2) No (3) (4) (5) 
covariates Quadratic Cubic  Quartic 
Enrollment in Gateway Math 
-0.209 -0.357 -0.341 0.089 
(0.223 (0.236) (0.225) (0.349) 
0.753** 0.896*** 0.896*** 0.897*** 
(0.247 (0.203) (0.202) (0.203) 
0.185 0.203 0.203 0.203 
(0.172 (0.174) (0.174) (0.174) 
Completion of Gateway Math 
-0.339 -0.684 -0.920* -0.596 
(0.294 (0.432) (0.411) (0.463) 
0.827** 0.943*** 0.948*** 0,950*** 
(0.296 (0.270) (0.268) (0.269) 
0.154 0.185 0.180 0.180 
(0.184 (0.181) (0.181) (0.180) 
4, +6] [-4,+6] [-4,+6] [4, +6] 
13,772 13,772 13,772 13,772 


(6) CER-optimal 


bandwidths 


0.255 
(0.321 
1.320*** 
(0.305 
-0.197 


(0.384, 


0.014 
(0.399 
1.546* 
(0.376 
-0.466 


0.371 
[-2, +3] 
5,073 


Results are from Difference-in-Regression Discontinuity models with alternate bandwidths and functional forms. 


**D < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10. 
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Table A5. Sensitivity analyses with interaction terms for the impact of assignment to upper DE 
(relative to college-level) on enrollment and completion of Intermediate Algebra (or higher) for 
nonexempt students. 
(1) No interaction (2) Interaction for DE*PERT (3) Interaction for DE*PERT and DE*PERT*POST 
Enrollment in Intermediate Algebra 


Upper DE -0.572*** -0.746*** -0.715*** 
0.107) (0.133) (0.121) 
Post 0.383*** 0.457*** 0.457*** 
0.104) (0.103) (0.102) 
Upper DE X Post -0.103 -0.126 -0.189 
0.093) (0.086) (0.144) 
Completion of Intermediate Algebra 
Upper DE -0.487*** -0.586*** -0.613*** 
0.092) (0.105) (0.124) 
Post 0.320** 0.318** 0.317** 
0.113) (0.113) (0.113) 
Upper DE X Post -0.069 -0.067 -0.185 
0.086) (0.085) (0.124) 
Bandwidth [-6, +5] [-6, +5] [-6, +5] 
N 8,968 8,968 8,968 


Results are from Difference-in-Regression Discontinuity models with different specifications of interaction terms. 
***D < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10. 


Table A6. Sensitivity analyses with interaction terms for the impact of assignment to upper DE 
(relative to college-level) on enrollment and completion of gateway math (or higher) for non- 
exempt students. 
(1) No interaction (2) Interaction for DE*PERT (3) Interaction for DE*PERT and DE*PERT*POST 
Enrollment in Gateway Algebra 


Upper DE -0.558*** -0.619*** -0.512*** 
0.104) (0.109) (0.154) 
Post 0.532*** 0.528*** 0.528*** 
0.105) (0.106) (0.106) 
Upper DE X Post 0.407*** 0.414*** 0.246 
0.084) (0.083) (0.172) 
Completion of Gateway Algebra 
Upper DE -0.579*** -0.61*** -0.449** 
0.116) (0.12) (0.189) 
Post 0.515*** 0.51*** 0:512*** 
0.100) (0.10) (0.100) 
Upper DE X Post 0.348*** 0.35*** 0.104 
0.084) (0.08) (0.196) 
Bandwidth [-6, +5] [-6, +5] [-6, +5] 
N 8,968 8,968 8,968 


Results are from Difference-in-Regression Discontinuity models with different specifications of interaction terms. 
***D < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10. 
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Table A7. Sensitivity analyses with interaction terms for the impact of assignment to lower DE 
(relative to upper DE) on enrollment and completion of Intermediate Algebra (or higher) for 
nonexempt students. 


(1) No interaction (2) Interaction for DE*PERT (3) Interaction for DE*PERT and DE*PERT*POST 


Upper DE 
Post 


Upper DE X Post 


Upper DE 
Post 
Upper DE X Post 


Bandwidth 
N 


-0.971*** 


Enrollment in Intermediate Algebra 


-0.887*** -0.798*** 
(0.163) (0.209) 
0.109 0.108 
(0.129) (0.129) 
0.720*** 0.568* 
(0.115) (0.238) 
Completion of Intermediate Algebra 
-0.673*** -0.582** 
(0.159) (0.236) 
0.436*** 0.436*** 
(0.108) (0.084) 
0.600*** 0.451 
(0.135) (0.279) 
[-4, +6] [-4, +6] 
13,772 13,772 


Results are from Difference-in-Regression Discontinuity models with different specifications of interaction terms. 
**D < 0.001, **p < 0.01, *p < 0.05, ~p < 0.10. 


Table A8. Sensitivity analyses with interaction terms for the impact of assignment to lower DE 
(relative to upper DE) on enrollment and completion of gateway math (or higher) for nonexempt 


students. 
(1) No interaction (2) Interaction for DE*PERT (3) Interaction for DE*PERT and DE*PERT*POST 
Enrollment in Gateway Algebra 
Upper DE -0.253 -0.324 -0.523 
0.213) (0.230) (0.424) 
Post 0.898*** 0.896*** 0.897*** 
0.204) (0.203) (0.203) 
Upper DE X Post 0.200 0.202 0.456 
0.175) (0.174) (0.452) 
Completion of Gateway Algebra 
Upper DE -0.375 -0.531 -0.386 
0.293) (0.392) (0.726) 
Post 0.948*** 0.944*** 0.944*** 
0.272) (0.180) (0.269) 
Upper DE X Post 0.173 0.062 -0.005 
0.183) (0.037) (0.607) 
Bandwidth [-4, +6] [-4, +6] [-4, +6] 
N 13,772 13,772 13,772 


Results are from Difference-in-Regression Discontinuity models with different specifications of interaction terms. 
***p < 0.001, *p < 0.01, *p < 0.05, ~p < 0.10. 


